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DETAILED ACTION 

Claim Objections 

1 . Claim 10 is objected to because of the following informalities: 

Claim 10, in the last line, limitation "the physical device" is unclear. 
Applicant refers to two physical devices in claim 10 and it is unclear which 
physical device applicant is referring to. 

Appropriate correction is required. 

Claim Rejections - 35 USC § 101 

1. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

2. Claims 36-45 are rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter. 

3. Claim 36 claims a computer data signal, which is functional descriptive material 
per-se. The claim does not define a tangible computer- readable medium on which the 
program is recorded (i.e. "When functional descriptive material is recorded on some 
computer-readable medium it becomes structurally and functionally interrelated to the 
medium and will be statutory in most cases since use of technology permits the function 
of the descriptive material to be realized"). 

A signal is neither a process ("actions"), nor machine, nor manufacture, nor 
composition of matter (i.e. a tangible "thing") and therefore does not fall within one of 
the four statutory categories of § 101. 
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Because the full scope of the claim as properly read in light of the disclosure 
encompasses non-statutory subject matter, the claim as a whole in non-statutory, under 
the present USPTO Interim Guidelines, 1300 Official Gazette Patent and Trademark 
Office 142 (Nov. 22, 2005). 

4. Claims 37-45 claim a machine readable medium embodying functional 
descriptive material. However, the claim does not define a computer-readable medium 
to be a memory/disk and is thus non-statutory for that reason (i.e. "When functional 
descriptive material is recorded on some computer-readable medium it becomes 
structurally and functionally interrelated to the medium and will be statutory in most 
cases since use of technology permits the function of the descriptive material to be 
realized"). 

However, the specification (at page 14, section 0043...) indicates "the storage 
medium can include, but is not limited to...", and therefore does not disqualify non- 
tangible storage mediums (e.g. carrier waves, infrared signals, digital signals, etc.) that 
data may be formatted (stored within the communication/transport medium, i.e. signal) 
for broadcasting. 

A "signal" embodying functional descriptive material is neither a process 
("actions"), nor machine, nor manufacture, nor composition of matter (i.e. a tangible 
"thing") and therefore does not fall within one of the four statutory categories of § 1 01 . 
Rather, "signal" is a form of energy, in the absence of any physical structure or tangible 
material. 
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Because the full scope of the claims as properly read in light of the disclosure 
encompasses non-statutory subject matter, the claim as a whole in non-statutory, under 
the present USPTO Interim Guidelines, 1300 Official Gazette Patent and Trademark 
Office 142 (Nov. 22, 2005). 

Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

6. Claims 15, 9-14, 16-17, and 36-46 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ludwig et al (US 7,1 85,054 B1 ) in view of Okuno (US 6,977,672 B1 ) 
and further in view of Saka (US 2004/0070608 A1 ). 

Regarding claim 1 , Ludwig teaches a method for exchanging information 
in a shared interactive environment (abstract) comprising: 
a first live video image and a second live video image (figure 40 element 203 
shows multiple live video images) 

Ludwig does not teach selecting a first physical device in a first live video 
image wherein the first physical device has information associated with it; 
causing the information to be transferred to a second physical device in a second 
live video image wherein the transfer is brought about by manipulating a visual 
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representation of the information; 

wherein the manipulation includes interacting with the first live video image and 
the second live video image; 

wherein the first physical device and the second physical device are part of the 
shared interactive environment; and 

wherein the first physical device and the second physical device are not the 
same. 

Okuno teaches an interactive environment that includes selecting a first 
physical device in a first live video image wherein the first physical device has 
information associated with it (figure 17 and column 4 lines 23-32, figure 18 and 
column 4 lines 35-41, figure 19 and column 4 lines 45-52,); 
wherein manipulation includes interacting with the live video images (column 4 
lines 38-41 , figure 1 8). It would have been obvious to one of ordinary skill in the 
art at the time the invention was made to incorporate Okuno's teaching of using 
remote camera control and a live video image of a device in order to remotely 
manipulate/interact with said device with Ludwig's videoconferencing invention 
for the benefit of allowing video conference participants to control computers, 
printers, and other commonly used conference devices that are not physically 
reachable. 

Ludwig in view of Okuno does not teach causing the information to be 
transferred to a second physical device in a second live video image wherein the 
transfer is brought about by manipulating a visual representation of the 
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information; 

wherein the first physical device and the second physical device are part of the 
shared interactive environment; and 

wherein the first physical device and the second physical device are not the 
same. 

Saka teaches causing information to be transferred to a second physical 
device wherein the transfer is brought about by manipulating a visual 
representation of the information (page 1 section 0011); 
wherein the first physical device and the second physical device are part of the 
shared interactive environment (page 1 section 001 1); and 
wherein the first physical device and the second physical device are not the 
same (page 1 section 001 1 ). It would have been obvious to one of ordinary skill 
in the art at the time the invention was made to use the information transfer 
interface taught by Saka in the videoconferencing with remote control of devices 
taught by Ludwig in view of Okuno for the benefit of providing videoconference 
participants with a convenient and simple (for participants) method of sharing 
information. For example, the action of sending data to the printer disclosed in 
Okuno (figure 10 shows an interface for controlling the printer, where the 
structure of the commands to carry out the actions are shown in figure 9 list 
element 7 and figure 5) from a computer as disclosed in Saka would be carried 
out using Saka's "drag and drop" method. Alternatively, Okuno's printer could be 
replaced with one of Saka's remote machines so that control of the multiple 
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computers disclosed in Saka's network could be initiated by the live video image 
manipulation taught by Okuno. 

Regarding claim 2, when read in light of claim 1 , Saka further teaches the 
manipulation is accomplished by dragging the representation from the first 
physical device and dropping the representation on the second physical device 
(abstract). 

Regarding claim 3, when read in light of claim 1 , limitation "changes to the 
physical devices are visible to all participants in the shared interactive 
environment" is inherent. If all participants in the videoconference have the 
capabilities disclosed in Ludwig in view of Okuno and Saka, all participants would 
have the ability to focus on any physical device and obtain current associated 
information, which would reflect any changes. 

Regarding claim 4, when read in light of claim 1, Okuno further teaches a 
physical device can include a printer (figure 18). In regards to limitation "a 
physical device can include a display, a projector, a facsimile machine, a 
personal digital assistant, a computer, and a portable computer", Okuno further 
discloses in column 5 lines 58-64 that commands can be executed on devices 
with addresses, such as IP addresses . It is well known in the art that displays, 
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projectors, facsimile machines, personal digital assistants, computers, and 
portable computers can have IP addresses associated with them. 

Regarding claim 5, when read in light of claim 1, Ludwig further teaches 
annotating at least one of the first live video image and the second live video 
image (fig 2b, col. 6 II. 57-61 . Updates/annotations are made to the image in real 
time). 

Regarding claim 9, when read in light of claim 1, Okuno further teaches at 
least one of the first physical device and the second physical device has 
associated with it a pop-up control panel through which a user can configure and 
control it (figure 18, column 4 lines 38-41). 

Regarding claim 10, Ludwig teaches a method for exchanging information 
in a shared interactive environment (abstract), comprising: 
a first live video image and a second live video image (figure 40 element 203 
shows multiple live video images). 

Ludwig does not teach selecting a first object wherein the first object is 
one of: 1 ) a physical device in a first live video image; and 2) an icon on a 
computing device; 

causing information associated with the first object to be transferred to a second 
object wherein the second object is the other of: 1 ) a physical device in a first live 
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video image; and 3) an icon on a computing device; 

wherein the transfer is brought about by manipulating a visual representation of 
the information; 

wherein the manipulation includes interacting with the first object and the second 
object; and 

wherein the physical device is part of the shared interactive environment. 

Okuno teaches selecting a first object wherein the first object is one of: 1 ) 
a physical device in a first live video image (figure 17 and column 4 lines 23-32, 
figure 18 and column 4 lines 35-41, figure 19 and column 4 lines 45-52); and 2) 
an icon on a computing device; 

wherein the manipulation includes interacting with the object in the live video 
(column 4 lines 38-41 , figure 1 8). It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to incorporate Okuno's teaching 
of using remote camera control and a live video image of a device in order to 
remotely manipulate/interact with said device with Ludwig's videoconferencing 
invention for the benefit of allowing video conference participants to control 
computers, printers, and other commonly used conference devices that are not 
physically reachable. 

Ludwig in view of Okuno does not teach causing information associated 
with the first object to be transferred to a second object wherein the second 
object is the other of: 1 ) a physical device in a first live video image; and 3) an 
icon on a computing device; 
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wherein the transfer is brought about by manipulating a visual representation of 
the information; 

wherein the manipulation includes interacting with the first object and the second 
object; and 

wherein the physical device is part of the shared interactive environment. 

Saka teaches causing information associated with the first object to be 
transferred to a second object wherein the second object is the other of: 1 ) a 
physical device in a first live video image; and 3) an icon on a computing device 
(page 1 section 001 1 ); 

wherein the transfer is brought about by manipulating a visual representation of 
the information (page 1 section 0011); 

wherein the manipulation includes interacting with the first object and the second 
object (page 1 section 001 1); and 

wherein the physical device is part of the shared interactive environment (page 1 
section 001 1 ). It would have been obvious to one of ordinary skill in the art at the 
time the invention was made to use the information transfer interface taught by 
Saka in the videoconferencing with remote control of devices taught by Ludwig in 
view of Okuno for the benefit of providing videoconference participants with a 
convenient and simple (for participants) method of sharing information. For 
example, the action of sending data to the printer disclosed in Okuno (figure 10 
shows an interface for controlling the printer, where the structure of the 
commands to carry out the actions are shown in figure 9 list element 7 and figure 
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5) from a computer as disclosed in Saka would be carried out using Saka's "drag 
and drop" method. Alternatively, Okuno's printer could be replaced with one of 
Saka's remote machines so that control of the multiple computers disclosed in 
Saka's network could be initiated by the live video image manipulation taught by 
Okuno. 

Regarding claim 1 1 , when read in light of claim 10, Saka further teaches 
the manipulation is accomplished by dragging the representation from the first 
object and dropping the representation on the second object (abstract). 

Regarding claim 12, when read in light of claim 10, limitation "changes to 
the physical device are visible to all participants in the shared interactive 
environment" is inherent. If all participants in the videoconference have the 
capabilities disclosed in Ludwig in view of Okuno and Saka, all participants would 
have the ability to focus on any physical device and obtain current associated 
information, which would reflect any changes. 

Regarding claim 13, when read in light of claim 10, Okuno further teaches 
a physical device can include a printer (figure 18). In regards to limitation "a 
physical device can include a display, a projector, a facsimile machine, a 
personal digital assistant, a computer, and a portable computer", Okuno further 
discloses in column 5 lines 58-64 that commands can be executed on devices 
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with addresses, such as IP addresses . It is well known in the art that displays, 
projectors, facsimile machines, personal digital assistants, computers, and 
portable computers can have IP addresses associated with them. 

Regarding claim 14, when read in light of claim 1 , Ludwig further teaches 
annotating the first live video image (fig 2b, col. 6 II. 57-61 . Updates/annotations 
are made to the image in real time). 

Regarding claim 16, when read in light of claim 10, Okuno further teaches 
the information can include a digital file (column 4 lines 25-29), a sound (column 
4 lines 29-32), and an audio/video presentation (column 4 lines 29-32). 

Regarding claim 17, when read in light of claim 10, Okuno further teaches 
the physical device has associated with it a pop-up control panel through which a 
user can configure and control it (figure 18, column 4 lines 38-41). 

Regarding claim 36, Ludwig, Okuno, and Saka all implement their 
inventions in hardware, which inherently requires machine readable instructions 
(i.e. a computer data signal), therefore a computer data signal embodying the 
method described in claim 1 is taught by Ludwig in view of Okuno and Saka (see 
analysis of claim 1). 
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Regarding claim 37, Ludwig, Okuno, and Saka all implement their 
inventions in hardware, which inherently requires machine readable instructions, 
therefore a computer data signal embodying the method described in claim 10 is 
taught by Ludwig in view of Okuno and Saka (see analysis of claim 1 0). 

Regarding claim 38, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 1 1 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 1 1 ). 

Regarding claim 39, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 12 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 1 2). 

Regarding claim 40, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 1 3 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 1 3). 
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Regarding claim 41, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 14 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 14). 

Regarding claim 42, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 15 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 1 5). 

Regarding claim 43, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 10 with the additional limitation of claim 7 is taught by 
Ludwig in view of Okuno and Saka (see analysis of claims 10 and 7). 

Regarding claim 44, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
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method described in claim 16 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 16). 

Regarding claim 45, when read in light of claim 37, Ludwig, Okuno, and 
Saka all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 17 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 1 7). 

Regarding claim 46 Ludwig, Okuno, and Saka all implement their 
inventions in hardware, and therefore teach a system with means for carrying out 
the method described in claim 1 is taught by Ludwig in view of Okuno and Saka 
(see analysis of claim 1 ). 

7. Claims 6, 8, and 15 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Ludwig et al (US 7,185,054 B1) in view of Okuno (US 6,977,672 B1) and Saka (US 
2004/0070608 A1 ), and further in view of Shneiderman (US 7,01 0,751 B2). 

Regarding claim 6, when read in light of claim 5, Ludwig in view of Okuno 
and Saka teaches the method of claim 5. 

Ludwig in view of Okuno and Saka does not teach automatically 
transferring the annotation to a physical device if the annotation is at least 
partially drawn over the physical device as it appears in a live video image. 
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Shneiderman teaches automatically transferring the annotation to an 
"image aspect" (a physical device would qualify as an image aspect, col. 7 II. 23- 
25) in an image if the annotation is at least partially drawn over the physical 
device as it appears in a live video image (col. 7 II. 29-32, col. 4 II. 27-29, see fig. 
5 annotation of Rae Earnshaw). It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to annotate aspects of images 
for the benefit of making annotations more specific to certain attributes of an 
image rather than the entire image. 

Regarding claim 8, Ludwig in view of Okuno and Saka further teaches the 
information can include a digital file (Okuno column 4 lines 25-29), a sound 
(Okuno column 4 lines 29-32), and an audio/video presentation (Okuno column 4 
lines 29-32). 

Ludwig in view of Okuno and Saka does not teach the information can 
include an annotation. 

Shneiderman teaches the information can include an annotation (page 3 
section 0055-0056). It would have been obvious to one of ordinary skill in the art 
at the time the invention was made to annotate aspects of images for the benefit 
of making annotations more specific to certain attributes of an image rather than 
the entire image. 
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Regarding claim 15, Ludwig in view of Okuno and Saka teaches the 
method of claim 14. 

Ludwig in view of Okuno and Saka does not teach automatically 
transferring the annotation to the physical device if the annotation is at least 
partially drawn over the physical device as it appears in a live video image. 

Shneiderman teaches automatically transferring the annotation to an 
"image aspect" (a physical device would qualify as an image aspect, col. 7 II. 23- 
25) in an image if the annotation is at least partially drawn over the physical 
device as it appears in a live video image (col. 7 II. 29-32, col. 4 II. 27-29, see fig. 
5 annotation of Rae Earnshaw). It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to annotate aspects of images 
for the benefit of making annotations more specific to certain attributes of an 
image rather than the entire image. 



8. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Ludwig et 
al (US 7,185,054 B1) in view of Okuno (US 6,977,672 B1) and Saka (US 2004/0070608 
A1 ), and further in view of Emens et al (US 6,463,343 B1 ). 



Regarding claim 7, when read in light of claim 1 , Ludwig in view of Okuno 
and Saka teach the method of claim 1 . 

Ludwig in view of Okuno and Saka does not teach that the first live video 
image and the second live video image are the same. 
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Limitation "the first live video image and the second live video image are 
the same", in light of claim 1 limitation "selecting a first physical device in a first 
live video image 

causing the information to be transferred to a second physical device in a second 
live video image wherein the transfer is brought about by manipulating a visual 
representation of the information" is taught by Emens column 1 lines 57-60 and 
figure 2c, in which a user can select a device to control amongst a plurality of 
devices in a single live image. It would have been obvious to one of ordinary skill 
in the art at the time the invention was made to incorporate Emen's teaching of 
being able to select a device to remotely control from a plurality of devices in a 
single live image with the invention taught by Ludwig in view of Okuno and Saka. 
This combination of inventive ideas would enable videoconference participants to 
remotely control devices in a room without adjusting the camera angle to zoom in 
on each device. 

9. Claims 18 and 21 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Ludwig et al (US 7,185,054 B1) in view of Shneiderman (US 7,010,751 B2). 



Regarding claim 18, Ludwig teaches a method for annotating a live video 
image wherein the annotation is visible to at least one participant in a shared 
interactive 
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environment (fig 2b, col. 6 II. 57-61 . Updates/annotations are made to the image 
in real time). 

Ludwig does not teach annotating the live video image and automatically 
transferring the annotation to a physical device if the annotation is at least 
partially drawn over the physical device as it appears in a live video image. 

Shneiderman teaches a method for annotating video image comprising 
annotating the live video image and automatically transferring the annotation to a 
physical device if the annotation is at least partially drawn over the physical 
device as it appears in a live video image (col. 7 II. 29-32, col. 4 II. 27-29, see fig. 
5 annotation of Rae Earnshaw). It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to provide the annotation 
capabilities of Shneiderman in the videoconferencing system and method taught 
by Ludwig for the benefit of easily and clearly communicating information about 
image aspects in a videoconferencing image to other participants. 

Regarding claim 21 , when read in light of claim 18, Ludwig in view of 
Shneiderman teaches the method of claim 18. 

Ludwig in view of Shneiderman does not teach the live video image is one 
of: a panoramic view and a zoomed view. 

Official notice is taken that panning and zooming are well known and 
widely used tools in video surveillance and conferencing systems. It would have 
been obvious to one of ordinary skill in the art at the time the invention was made 
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to include panning and zooming capabilities in the videoconferencing system 
taught by Ludwig in view of Shneiderman for the benefit of allowing 
videoconference participants to have access to view of any resources in the 
collective rooms, such as whiteboards or illustrative models. 

10. Claims 19-20 and 22-23 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ludwig et al (US 7,1 85,054 B1 ) in view of Shneiderman (US 
7,01 0,751 B2), and further in view of Okuno (US 6,977,672 B1 ). 

Regarding claim 19, when read in light of claim 18, Ludwig in view of 
Shneiderman teaches the method of claim 18. 

Ludwig in view of Shneiderman does not teach at least one participant can 
interact with the physical device. 

Okuno teaches an video in which a viewed physical devices can be controlled 
remotely by a participant (column 4 lines 38-41 , figure 18). It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to 
incorporate Okuno's teaching of using remote camera control and a live video 
image of a device in order to remotely manipulate/interact with devices that are 
not physically reachable. 
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Regarding claim 20, when read in light of claim 1 8, Ludwig further teaches 
each of the at least one participants can interact with the shared interactive 
environment through different computing devices (col. 6 II. 28-51). 



Regarding claim 22, when read in light of claim 18, Ludwig in view of 
Shneiderman and Okuno further teaches the physical device can include a 
display, a projector, a printer, a facsimile machine, a personal digital assistant, a 
computer, and a portable computer. Okuno discloses in column 5 lines 58-64 
that commands can be executed on devices with addresses, such as IP 
addresses. It is well known in the art that displays, projectors, facsimile 
machines, personal digital assistants, computers, and portable computers can 
have IP addresses associated with them. 



Regarding claim 23, when read in light of claim 18, Ludwig in view of 
Shneiderman and Okuno further teaches the physical device has associated with 
it a pop-up control panel through which a user can configure and control it 
(Okuno figure 18, column 4 lines 38-41). 



1 1 . Claims 24, 26-27, 30, 32, and 34-35 are rejected under 35 U.S.C. 1 03(a) as 
being unpatentable over Emens et al (US 6,463,343 B1) in view of Ayatsuka et al (US 
7,188,139 B1), and further in view of Burt et al. "Object tracking with a moving camera", 
IEEE An Application of Dynamic Motion Analysis. 1989 pp. 2-12. 
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Regarding claim 24, Emens teaches a shared interactive environment, 
comprising: 

a camera system to provide a first live view and second live view (col. 3 II. 8-9, 
where the first live view and second live view are the same in this instance); 
a first graphical user interface (GUI) coupled to the camera system and to 
present the first live view and the second live view, wherein the views can 
capture a physical device (abstract); 

a device controller to dynamically control the physical device in response to 
interaction of a first user with the GUI (fig. 2e, col. 3 II. 10-11). 

Emens does not teach the second live view can be configured to zoom in 
on a portion of the first live view; 

a device tracker coupled to the camera system and to dynamically recognize new 
physical devices; and 

wherein the camera system can be mounted on a mobile, robotic platform. 

In regards to limitation "wherein the second live view can be configured to 
zoom in on a portion of the first live view", official notice is taken that zooming is 
a well known and widely used tool in video surveillance and conferencing 
systems. It would have been obvious to one of ordinary skill in the art at the time 
the invention was made to include panning and zooming capabilities in the 
interactive environment taught by Emens for the benefit of allowing 
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videoconference participants to have access to view of any resources in the 
collective rooms, such as whiteboards or illustrative models. 

Ayatsuka teaches a camera connected to a computer that can detect and 
recognize devices in the camera's view, and allows the viewed device to be 
controlled by the computer (abstract). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to incorporate the 
device detection capabilities taught by Ayatsuka in the device control system 
taught by Emens for the benefit of allowing a user to register devices for control 
by a computer system without manually entering information for each device. 

Burt teaches a robotic camera that detects and tracks a target (abstract). 
It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to incorporate Burt's teaching of a robotic camera that keeps 
targeted objects in its line of view in the interactive environment taught by Emens 
in view of Ayatsuka for the benefit of maintaining a view of devices that the user 
may want to exercise control over. Burt's abstract teaches the application of this 
concept to automated surveillance in the abstract. 

Regarding claim 26, when read in light of claim 24, Emens further teaches 
the device controller can control the physical device through at least one of: 1) an 
infrared communication channel; and 2) one or more networks (fig. 1 col. 5 II. 26- 
31). 
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Regarding claim 27, when read in light of claim 24, Emens in view of 
Ayatsuka and Burt further teaches the device tracker can recognize new physical 
devices by at least one of: 1 ) image pattern recognition (Burt pg. 3 col. 2 
paragraph 2, fig. 2); 2) radio frequency transmission; and 3) acoustic signal. 

Regarding claim 30, when read in light of claim 24, Emens further teaches 
the GUI is implemented as one or more web pages (col. 2 II. 60-65). 

Regarding claim 32, when read in light of claim 24, Emens further teaches 
the physical device has a pop-up control panel that can be made apparent to the 
first user through the first GUI and wherein the pop-up control panel allows the 
first user to control and configure the physical device (fig. 2e). 

Regarding claim 34, when read in light of claim 24, both Emens and 
Ayatsuka teach software based control schemes for controlling objects and 
devices. It is inherent that the physical device can be represented by a set of 
attributes and a set of behaviors, as those are the only ways of representing an 
object to a computer program. 

Regarding claim 35, when read in light of claim 34, Emens in view of 
Ayatsuka and Burt teach the shared interactive environment of claim 34. 
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Emens in view of Ayatsuka and Burt does not teach the representation of 
the physical device is part of a device hierarchy. 

Both Emens and Ayatsuka teach software based control schemes for 
controlling objects and devices. Object oriented programming is an obvious 
approach to developing such software, as each device has its own attributes and 
control routines. The concept of classes and inheritance are well known and 
often used to create programs that support a variety of objects. Official notice is 
taken that it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to use objects and inheritance to implement device 
control software for the benefit of optimizing the software and reducing 
redundancy in the application programming for devices that may share similar 
functions. 

12. Claim 25 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et al (US 6,463,343 B1) in view of Ayatsuka et al (US 7,188,139 B1) and Burt et al. 
"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
Analysis. 1 989 pp. 2-12, and further in view of Andersson (US 2002/01 1 1 999 A1 ). 

Regarding claim 25, when read in light of claim 24, Emens in view of 
Ayatsuka and Burt teaches the interactive environment of claim 24 and suggests 
implementing device control in a video teleconferencing environment (fig. 5), but 
does not disclose multiple users having access to the system (i.e. the first GUI 
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allows the first user to interact the physical device; and wherein the interaction of 
the first user is apparent to a second user via a second GUI). 

Andersson teaches a system that allows networked computers to access 
and control devices connected to any computer on that network, where both host 
and remote computers can monitor device events (pg. 2 sect. 0019, 0023). It 
would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the remote access and device monitoring 
capabilities taught by Andersson in the shared interactive environment taught by 
Emens in view of Ayatsuka for the benefit of enhancing collaboration and making 
a more realistic environment for users who may share resources. 

13. Claim 28 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et al (US 6,463,343 B1) in view of Ayatsuka et al (US 7,188,139 B1) and Burt et al. 
"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
Analysis. 1 989 pp. 2-1 2, and further in view of Hildebrandt (US 2004/007061 6). 

Regarding claim 28, when read in light of claim 24, Emens in view of 
Ayatsuka and Burt teaches the interactive environment of claim 24, wherein the 
physical device can be a display (Ayatsuka fig. 5, col. 22 II. 63-67). 
Emens in view of Ayatsuka and Burt does not teach the display can include an 
image stack. 
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Hildebrandt teaches an archive memory for use with an electronic 
whiteboard in which images associated with the whiteboard are grouped and 
saved. These images are stored as a stack, where the oldest image is deleted 
when a new image is added (see fig. 22 el 524). It would have been obvious to 
one of ordinary skill in the art at the time the invention was made to store a 
plurality of images captured from a display for the benefit of allowing a user to 
continually use a display resource while stil being able to access data previously 
represented on that display. 

14. Claim 33 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et al (US 6,463,343 B1) in view of Ayatsuka et al (US 7,188,139 B1) and Burt et al. 
"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
Analysis. 1989 pp. 2-12, and further in view of Vetterli (US 2002/0075282). 

Regarding claim 33, when read in light of claim 24, Emens in view of 
Ayatsuka and Burt teaches the shared interactive environment of claim 24. 

Emens in view of Ayatsuka and Burt does not teach the user can annotate 
at least one of: 1 ) the first live view; and 2) the second live 
view. 

Ayatsuka suggests one environment in which devices remotely controlled 
through a camera view could be used is in a teleconference (fig. 5). 
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Vetterli teaches a teleconference environment in which the user can 
annotate a live view (pg. 3 sect 0055). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to include the 
annotation abilities taught by Vetterli in the interactive environment taught by 
Emens in view of Ayatsuka and Burt for the benefit of training and guidance, and 
for information services, augmented reality, shopping, and meeting 
enhancement, as suggested by Vetterli. 

15. Claim 29 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et al (US 6,463,343 B1) in view of Ayatsuka et al (US 7,188,139 B1) and Burt et al. 
"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
Analysis. 1 989 pp. 2-12, and Andersson (US 2002/01 1 1 999 A1 ), and further in view of 
Westfield (US 6,677,979 B1). 

Regarding claim 29, when read in light of claim 25, Emens in view of 
Ayatsuka, Burt, and Andersson teaches the shared interactive environment of 
claim 25. 

Emens in view of Ayatsuka, Burt, and Andersson does not teach the first 
GUI can provide a second live view that is different from the second live view 
provided by the second GUI. 

Ayatsuka suggests one environment in which devices remotely controlled 
through a camera view could be used is in a teleconference (fig. 5). 
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Westfield teaches a teleconference environment in which a participant can 
see 2 views, where the second view can be specified for each participant 
(abstract, fig 8b, fig 7 and col. 2 II. 46-56). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to include the multiple 
views of an environment taught by Westfield in the interactive environment taught 
by Emens in view of Ayatsuka, Burt, and Andersson for the benefit of more 
closely mimicking human vision by providing both wide angle and focused views 
(col. 1 II. 43-54). 

16. Claim 31 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et al (US 6,463,343 B1) in view of Ayatsuka et al (US 7,188,139 B1) and Burt et al. 
"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
Analysis. 1989 pp. 2-12, and further in view of Westfield (US 6,677,979 B1). 

Regarding claim 31 , when read in light of claim 24, Emens in view of 
Ayatsuka and Burt teaches the shared interactive environment of claim 24. 

Emens in view of Ayatsuka and Burt does not teach the first user can select the second 

live view by drawing a diagonal in the first live view. 

Westfield teaches a teleconference environment in which a participant can see 2 views, 
where the second view can be specified by a participant dragging a rectangle across 
the image in the first view, (abstract, fig 8b, fig 7 and col. 6 II. 38-40). It would have 
been obvious to one of ordinary skill in the art at the time the invention was made to 
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include the zoomed view of an environment taught by Westfield in the interactive 
environment taught by Emens in view of Ayatsuka, Burt, and Andersson for the benefit 
of more closely mimicking human vision by providing both wide angle and focused 
views (col. 1 II. 43-54). 



Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JASMINE STOKELY-COLLINS whose telephone 
number is (571 ) 270-3459. The examiner can normally be reached on M-Th 9:30-5:00 
EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Andrew Koenig can be reached on (571) 272-7296. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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